{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "from exp.nb_02 import *\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initial setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 9 video](https://course19.fast.ai/videos/?lesson=9&t=1786)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mpl.rcParams['image.cmap'] = 'gray'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train,y_train,x_valid,y_valid = get_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n,m = x_train.shape\n",
    "c = y_train.max()+1\n",
    "nh = 50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Model(nn.Module):\n",
    "    def __init__(self, n_in, nh, n_out):\n",
    "        super().__init__()\n",
    "        self.layers = [nn.Linear(n_in,nh), nn.ReLU(), nn.Linear(nh,n_out)]\n",
    "        \n",
    "    def __call__(self, x):\n",
    "        for l in self.layers: x = l(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model(m, nh, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = model(x_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross entropy loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we will need to compute the softmax of our activations. This is defined by:\n",
    "\n",
    "$$\\hbox{softmax(x)}_{i} = \\frac{e^{x_{i}}}{e^{x_{0}} + e^{x_{1}} + \\cdots + e^{x_{n-1}}}$$\n",
    "\n",
    "or more concisely:\n",
    "\n",
    "$$\\hbox{softmax(x)}_{i} = \\frac{e^{x_{i}}}{\\sum_{0 \\leq j \\leq n-1} e^{x_{j}}}$$ \n",
    "\n",
    "In practice, we will need the log of the softmax when we calculate the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def log_softmax(x): return (x.exp()/(x.exp().sum(-1,keepdim=True))).log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm_pred = log_softmax(pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cross entropy loss for some target $x$ and some prediction $p(x)$ is given by:\n",
    "\n",
    "$$ -\\sum x\\, \\log p(x) $$\n",
    "\n",
    "But since our $x$s are 1-hot encoded, this can be rewritten as $-\\log(p_{i})$ where i is the index of the desired target."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be done using numpy-style [integer array indexing](https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.indexing.html#integer-array-indexing). Note that PyTorch supports all the tricks in the advanced indexing methods discussed in that link."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([5, 0, 4])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-2.2674, -2.1714, -2.3043], grad_fn=<IndexBackward>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sm_pred[[0,1,2], [5,0,4]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50000"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 9 video](https://course19.fast.ai/videos/?lesson=9&t=2081)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def nll(input, target): return -input[range(target.shape[0]), target].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = nll(sm_pred, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(2.3019, grad_fn=<NegBackward>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the formula \n",
    "\n",
    "$$\\log \\left ( \\frac{a}{b} \\right ) = \\log(a) - \\log(b)$$ \n",
    "\n",
    "gives a simplification when we compute the log softmax, which was previously defined as `(x.exp()/(x.exp().sum(-1,keepdim=True))).log()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def log_softmax(x): return x - x.exp().sum(-1,keepdim=True).log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_near(nll(log_softmax(pred), y_train), loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, there is a way to compute the log of the sum of exponentials in a more stable way, called the [LogSumExp trick](https://en.wikipedia.org/wiki/LogSumExp). The idea is to use the following formula:\n",
    "\n",
    "$$\\log \\left ( \\sum_{j=1}^{n} e^{x_{j}} \\right ) = \\log \\left ( e^{a} \\sum_{j=1}^{n} e^{x_{j}-a} \\right ) = a + \\log \\left ( \\sum_{j=1}^{n} e^{x_{j}-a} \\right )$$\n",
    "\n",
    "where a is the maximum of the $x_{j}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def logsumexp(x):\n",
    "    m = x.max(-1)[0]\n",
    "    return m + (x-m[:,None]).exp().sum(-1).log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This way, we will avoid an overflow when taking the exponential of a big activation. In PyTorch, this is already implemented for us. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_near(logsumexp(pred), pred.logsumexp(-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we can use it for our `log_softmax` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def log_softmax(x): return x - x.logsumexp(-1,keepdim=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_near(nll(log_softmax(pred), y_train), loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then use PyTorch's implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_near(F.nll_loss(F.log_softmax(pred, -1), y_train), loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In PyTorch, `F.log_softmax` and `F.nll_loss` are combined in one optimized function, `F.cross_entropy`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_near(F.cross_entropy(pred, y_train), loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic training loop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basically the training loop repeats over the following steps:\n",
    "- get the output of the model on a batch of inputs\n",
    "- compare the output to the labels we have and compute a loss\n",
    "- calculate the gradients of the loss with respect to every parameter of the model\n",
    "- update said parameters with those gradients to make them a little bit better"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 9 video](https://course19.fast.ai/videos/?lesson=9&t=2542)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_func = F.cross_entropy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def accuracy(out, yb): return (torch.argmax(out, dim=1)==yb).float().mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 0.0154,  0.2477, -0.0076, -0.1921, -0.0089,  0.0422,  0.0218, -0.0388,\n",
       "         -0.0303, -0.0328], grad_fn=<SelectBackward>), torch.Size([64, 10]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bs=64                  # batch size\n",
    "\n",
    "xb = x_train[0:bs]     # a mini-batch from x\n",
    "preds = model(xb)      # predictions\n",
    "preds[0], preds.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(2.3076, grad_fn=<NllLossBackward>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yb = y_train[0:bs]\n",
    "loss_func(preds, yb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.1719)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(preds, yb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 0.5   # learning rate\n",
    "epochs = 1 # how many epochs to train for"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for epoch in range(epochs):\n",
    "    for i in range((n-1)//bs + 1):\n",
    "#         set_trace()\n",
    "        start_i = i*bs\n",
    "        end_i = start_i+bs\n",
    "        xb = x_train[start_i:end_i]\n",
    "        yb = y_train[start_i:end_i]\n",
    "        loss = loss_func(model(xb), yb)\n",
    "\n",
    "        loss.backward()\n",
    "        with torch.no_grad():\n",
    "            for l in model.layers:\n",
    "                if hasattr(l, 'weight'):\n",
    "                    l.weight -= l.weight.grad * lr\n",
    "                    l.bias   -= l.bias.grad   * lr\n",
    "                    l.weight.grad.zero_()\n",
    "                    l.bias  .grad.zero_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.3465, grad_fn=<NllLossBackward>), tensor(0.9375))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss_func(model(xb), yb), accuracy(model(xb), yb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using parameters and optim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `nn.Module.__setattr__` and move relu to functional:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 9 video](https://course19.fast.ai/videos/?lesson=9&t=2818)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Model(nn.Module):\n",
    "    def __init__(self, n_in, nh, n_out):\n",
    "        super().__init__()\n",
    "        self.l1 = nn.Linear(n_in,nh)\n",
    "        self.l2 = nn.Linear(nh,n_out)\n",
    "        \n",
    "    def __call__(self, x): return self.l2(F.relu(self.l1(x)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model(m, nh, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "l1: Linear(in_features=784, out_features=50, bias=True)\n",
      "l2: Linear(in_features=50, out_features=10, bias=True)\n"
     ]
    }
   ],
   "source": [
    "for name,l in model.named_children(): print(f\"{name}: {l}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Model(\n",
       "  (l1): Linear(in_features=784, out_features=50, bias=True)\n",
       "  (l2): Linear(in_features=50, out_features=10, bias=True)\n",
       ")"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear(in_features=784, out_features=50, bias=True)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.l1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit():\n",
    "    for epoch in range(epochs):\n",
    "        for i in range((n-1)//bs + 1):\n",
    "            start_i = i*bs\n",
    "            end_i = start_i+bs\n",
    "            xb = x_train[start_i:end_i]\n",
    "            yb = y_train[start_i:end_i]\n",
    "            loss = loss_func(model(xb), yb)\n",
    "\n",
    "            loss.backward()\n",
    "            with torch.no_grad():\n",
    "                for p in model.parameters(): p -= p.grad * lr\n",
    "                model.zero_grad()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.1094, grad_fn=<NllLossBackward>), tensor(0.9375))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fit()\n",
    "loss_func(model(xb), yb), accuracy(model(xb), yb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Behind the scenes, PyTorch overrides the `__setattr__` function in `nn.Module` so that the submodules you define are properly registered as parameters of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DummyModule():\n",
    "    def __init__(self, n_in, nh, n_out):\n",
    "        self._modules = {}\n",
    "        self.l1 = nn.Linear(n_in,nh)\n",
    "        self.l2 = nn.Linear(nh,n_out)\n",
    "        \n",
    "    def __setattr__(self,k,v):\n",
    "        if not k.startswith(\"_\"): self._modules[k] = v\n",
    "        super().__setattr__(k,v)\n",
    "        \n",
    "    def __repr__(self): return f'{self._modules}'\n",
    "    \n",
    "    def parameters(self):\n",
    "        for l in self._modules.values():\n",
    "            for p in l.parameters(): yield p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'l1': Linear(in_features=784, out_features=50, bias=True), 'l2': Linear(in_features=50, out_features=10, bias=True)}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mdl = DummyModule(m,nh,10)\n",
    "mdl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[torch.Size([50, 784]),\n",
       " torch.Size([50]),\n",
       " torch.Size([10, 50]),\n",
       " torch.Size([10])]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[o.shape for o in mdl.parameters()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Registering modules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the original `layers` approach, but we have to register the modules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 9 video](https://course19.fast.ai/videos/?lesson=9&t=2997)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layers = [nn.Linear(m,nh), nn.ReLU(), nn.Linear(nh,10)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Model(nn.Module):\n",
    "    def __init__(self, layers):\n",
    "        super().__init__()\n",
    "        self.layers = layers\n",
    "        for i,l in enumerate(self.layers): self.add_module(f'layer_{i}', l)\n",
    "        \n",
    "    def __call__(self, x):\n",
    "        for l in self.layers: x = l(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model(layers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Model(\n",
       "  (layer_0): Linear(in_features=784, out_features=50, bias=True)\n",
       "  (layer_1): ReLU()\n",
       "  (layer_2): Linear(in_features=50, out_features=10, bias=True)\n",
       ")"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nn.ModuleList"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`nn.ModuleList` does this for us."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 9 video](https://course19.fast.ai/videos/?lesson=9&t=3173)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SequentialModel(nn.Module):\n",
    "    def __init__(self, layers):\n",
    "        super().__init__()\n",
    "        self.layers = nn.ModuleList(layers)\n",
    "        \n",
    "    def __call__(self, x):\n",
    "        for l in self.layers: x = l(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = SequentialModel(layers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Model(\n",
       "  (layers): ModuleList(\n",
       "    (0): Linear(in_features=784, out_features=50, bias=True)\n",
       "    (1): ReLU()\n",
       "    (2): Linear(in_features=50, out_features=10, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.2131, grad_fn=<NllLossBackward>), tensor(0.9375))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fit()\n",
    "loss_func(model(xb), yb), accuracy(model(xb), yb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nn.Sequential"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`nn.Sequential` is a convenient class which does the same as the above:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 9 video](https://course19.fast.ai/videos/?lesson=9&t=3199)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = nn.Sequential(nn.Linear(m,nh), nn.ReLU(), nn.Linear(nh,10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.2167, grad_fn=<NllLossBackward>), tensor(0.9375))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fit()\n",
    "loss_func(model(xb), yb), accuracy(model(xb), yb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn.Sequential??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): Linear(in_features=784, out_features=50, bias=True)\n",
       "  (1): ReLU()\n",
       "  (2): Linear(in_features=50, out_features=10, bias=True)\n",
       ")"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### optim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's replace our previous manually coded optimization step:\n",
    "\n",
    "```python\n",
    "with torch.no_grad():\n",
    "    for p in model.parameters(): p -= p.grad * lr\n",
    "    model.zero_grad()\n",
    "```\n",
    "\n",
    "and instead use just:\n",
    "\n",
    "```python\n",
    "opt.step()\n",
    "opt.zero_grad()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 9 video](https://course19.fast.ai/videos/?lesson=9&t=3278)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Optimizer():\n",
    "    def __init__(self, params, lr=0.5): self.params,self.lr=list(params),lr\n",
    "        \n",
    "    def step(self):\n",
    "        with torch.no_grad():\n",
    "            for p in self.params: p -= p.grad * self.lr\n",
    "\n",
    "    def zero_grad(self):\n",
    "        for p in self.params: p.grad.data.zero_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = nn.Sequential(nn.Linear(m,nh), nn.ReLU(), nn.Linear(nh,10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt = Optimizer(model.parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for epoch in range(epochs):\n",
    "    for i in range((n-1)//bs + 1):\n",
    "        start_i = i*bs\n",
    "        end_i = start_i+bs\n",
    "        xb = x_train[start_i:end_i]\n",
    "        yb = y_train[start_i:end_i]\n",
    "        pred = model(xb)\n",
    "        loss = loss_func(pred, yb)\n",
    "\n",
    "        loss.backward()\n",
    "        opt.step()\n",
    "        opt.zero_grad()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.0797, grad_fn=<NllLossBackward>), tensor(0.9375))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss,acc = loss_func(model(xb), yb), accuracy(model(xb), yb)\n",
    "loss,acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch already provides this exact functionality in `optim.SGD` (it also handles stuff like momentum, which we'll look at later - except we'll be doing it in a more flexible way!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "from torch import optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optim.SGD.step??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_model():\n",
    "    model = nn.Sequential(nn.Linear(m,nh), nn.ReLU(), nn.Linear(nh,10))\n",
    "    return model, optim.SGD(model.parameters(), lr=lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(2.3222, grad_fn=<NllLossBackward>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model,opt = get_model()\n",
    "loss_func(model(xb), yb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for epoch in range(epochs):\n",
    "    for i in range((n-1)//bs + 1):\n",
    "        start_i = i*bs\n",
    "        end_i = start_i+bs\n",
    "        xb = x_train[start_i:end_i]\n",
    "        yb = y_train[start_i:end_i]\n",
    "        pred = model(xb)\n",
    "        loss = loss_func(pred, yb)\n",
    "\n",
    "        loss.backward()\n",
    "        opt.step()\n",
    "        opt.zero_grad()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.0436, grad_fn=<NllLossBackward>), tensor(1.))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss,acc = loss_func(model(xb), yb), accuracy(model(xb), yb)\n",
    "loss,acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Randomized tests can be very useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 9 video](https://course19.fast.ai/videos/?lesson=9&t=3442)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert acc>0.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset and DataLoader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's clunky to iterate through minibatches of x and y values separately:\n",
    "\n",
    "```python\n",
    "    xb = x_train[start_i:end_i]\n",
    "    yb = y_train[start_i:end_i]\n",
    "```\n",
    "\n",
    "Instead, let's do these two steps together, by introducing a `Dataset` class:\n",
    "\n",
    "```python\n",
    "    xb,yb = train_ds[i*bs : i*bs+bs]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 9 video](https://course19.fast.ai/videos/?lesson=9&t=3578)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class Dataset():\n",
    "    def __init__(self, x, y): self.x,self.y = x,y\n",
    "    def __len__(self): return len(self.x)\n",
    "    def __getitem__(self, i): return self.x[i],self.y[i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ds,valid_ds = Dataset(x_train, y_train),Dataset(x_valid, y_valid)\n",
    "assert len(train_ds)==len(x_train)\n",
    "assert len(valid_ds)==len(x_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[0., 0., 0.,  ..., 0., 0., 0.],\n",
       "         [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "         [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "         [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "         [0., 0., 0.,  ..., 0., 0., 0.]]), tensor([5, 0, 4, 1, 9]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xb,yb = train_ds[0:5]\n",
    "assert xb.shape==(5,28*28)\n",
    "assert yb.shape==(5,)\n",
    "xb,yb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model,opt = get_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for epoch in range(epochs):\n",
    "    for i in range((n-1)//bs + 1):\n",
    "        xb,yb = train_ds[i*bs : i*bs+bs]\n",
    "        pred = model(xb)\n",
    "        loss = loss_func(pred, yb)\n",
    "\n",
    "        loss.backward()\n",
    "        opt.step()\n",
    "        opt.zero_grad()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.4507, grad_fn=<NllLossBackward>), tensor(0.8750))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss,acc = loss_func(model(xb), yb), accuracy(model(xb), yb)\n",
    "assert acc>0.7\n",
    "loss,acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataLoader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously, our loop iterated over batches (xb, yb) like this:\n",
    "\n",
    "```python\n",
    "for i in range((n-1)//bs + 1):\n",
    "    xb,yb = train_ds[i*bs : i*bs+bs]\n",
    "    ...\n",
    "```\n",
    "\n",
    "Let's make our loop much cleaner, using a data loader:\n",
    "\n",
    "```python\n",
    "for xb,yb in train_dl:\n",
    "    ...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 9 video](https://course19.fast.ai/videos/?lesson=9&t=3674)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DataLoader():\n",
    "    def __init__(self, ds, bs): self.ds,self.bs = ds,bs\n",
    "    def __iter__(self):\n",
    "        for i in range(0, len(self.ds), self.bs): yield self.ds[i:i+self.bs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dl = DataLoader(train_ds, bs)\n",
    "valid_dl = DataLoader(valid_ds, bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xb,yb = next(iter(valid_dl))\n",
    "assert xb.shape==(bs,28*28)\n",
    "assert yb.shape==(bs,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(3)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADXNJREFUeJzt3X+IVXUax/HPUxlpSlmSqU3ZjrHsEk1uQ2wUS7UY7SJYgVHQMmuyU1Cw1RYbQ1AUgSzbj6U/DKNBo99pbVK2GhHbryW0H5Rl2Q9cNceZNSOVijCf/WPOxGhzv/fOvefcc8fn/QK5957nnnMeLn7mnHu/596vubsAxHNI2Q0AKAfhB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8Q1GHN3JmZcTkhUDB3t1qe19CR38wuNLOPzexTM7u5kW0BaC6r99p+MztU0kZJcyRtlbRW0uXu/mFiHY78QMGaceQ/U9Kn7v65u38v6XFJ8xrYHoAmaiT8MyRtGfZ4a7ZsP2bWbWbrzGxdA/sCkLNGPvAb6dTiJ6f17r5E0hKJ036glTRy5N8qqW3Y4xMkbWusHQDN0kj410o6xcxONrPDJV0maWU+bQEoWt2n/e6+18yulbRa0qGSet39g9w6A1Couof66toZ7/mBwjXlIh8AYxfhB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0E1dYpu1KejoyNZv/766yvW2tvbk+tOmDAhWe/p6UnWjzrqqGT9hRdeqFjbvXt3cl0UiyM/EBThB4Ii/EBQhB8IivADQRF+ICjCDwTV0Cy9ZrZJ0m5JP0ja6+6dVZ7PLL0jmDhxYrK+efPmZP3oo4/Os51cffHFFxVrqesTJGn58uV5txNCrbP05nGRz3nuviOH7QBoIk77gaAaDb9LWmNmb5lZdx4NAWiORk/7z3b3bWZ2nKQXzewjd39l+BOyPwr8YQBaTENHfnfflt0OSHpG0pkjPGeJu3dW+zAQQHPVHX4zO9LMJg3dl3SBpPV5NQagWI2c9k+V9IyZDW3nUXf/Vy5dAShcQ+P8o94Z4/wjmjRpUrK+atWqZP3LL7+sWHvnnXeS686ePTtZP+mkk5L1tra2ZH38+PEVa/39/cl1zzrrrGS92vpR1TrOz1AfEBThB4Ii/EBQhB8IivADQRF+ICiG+tCQKVOmJOs33XRTXTVJWrBgQbK+bNmyZD0qhvoAJBF+ICjCDwRF+IGgCD8QFOEHgiL8QFBM0Y2G7NiR/uHm119/vWKt2jh/ta8bM87fGI78QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4/xoyOTJk5P1np6eurc9ffr0utdFdRz5gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiCoqr/bb2a9kuZKGnD3U7Nlx0h6QtJMSZskXeruX1XdGb/bP+Z0dHQk60899VSyPmvWrIq1jRs3JtedM2dOsr5ly5ZkPao8f7d/qaQLD1h2s6SX3P0USS9ljwGMIVXD7+6vSNp5wOJ5koZ+RmWZpIty7gtAwep9zz/V3fskKbs9Lr+WADRD4df2m1m3pO6i9wNgdOo98veb2TRJym4HKj3R3Ze4e6e7d9a5LwAFqDf8KyV1Zfe7JD2bTzsAmqVq+M3sMUn/kfRzM9tqZgslLZI0x8w+kTQnewxgDKk6zp/rzhjnbzldXV3J+u23356st7W1JevffvttxdrcuXOT67788svJOkaW5zg/gIMQ4QeCIvxAUIQfCIrwA0ERfiAofrr7IDBx4sSKtRtvvDG57i233JKsH3JI+viwc+eB3/na3znnnFOx9tFHHyXXRbE48gNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIzzHwSWLl1asXbJJZc0tO3ly5cn6/fee2+yzlh+6+LIDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBMc5/EGhvby9s24sXL07W33jjjcL2jWJx5AeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoKqO85tZr6S5kgbc/dRs2W2S/iTpf9nTetx9VVFNIm3NmjUVax0dHYVtW6p+HcCiRYsq1rZt21ZXT8hHLUf+pZIuHGH5Pe5+evaP4ANjTNXwu/srktLTsgAYcxp5z3+tmb1nZr1mNjm3jgA0Rb3hXyypXdLpkvok3VXpiWbWbWbrzGxdnfsCUIC6wu/u/e7+g7vvk/SApDMTz13i7p3u3llvkwDyV1f4zWzasIcXS1qfTzsAmqWWob7HJJ0raYqZbZV0q6Rzzex0SS5pk6SrCuwRQAHM3Zu3M7Pm7SyQ8ePHV6w9/PDDyXXPOOOMZP3EE0+sq6ch27dvr1hbsGBBct3Vq1c3tO+o3N1qeR5X+AFBEX4gKMIPBEX4gaAIPxAU4QeCYqjvIHfEEUck64cdlr7UY9euXXm2s5/vvvsuWb/hhhuS9fvvvz/Pdg4aDPUBSCL8QFCEHwiK8ANBEX4gKMIPBEX4gaAY50fSaaedlqzfc889yfp5551X9743b96crM+cObPubR/MGOcHkET4gaAIPxAU4QeCIvxAUIQfCIrwA0Exzt8CJkyYkKx/8803Tepk9CZPTk/T2NvbW7E2b968hvY9Y8aMZL2vr6+h7Y9VjPMDSCL8QFCEHwiK8ANBEX4gKMIPBEX4gaDSP9ouyczaJD0k6XhJ+yQtcfd/mNkxkp6QNFPSJkmXuvtXxbU6drW3tyfrr732WrL+/PPPJ+vr16+vWKs21r1w4cJkfdy4ccl6tbH2WbNmJespn332WbIedRw/L7Uc+fdK+ou7/0LSryVdY2a/lHSzpJfc/RRJL2WPAYwRVcPv7n3u/nZ2f7ekDZJmSJonaVn2tGWSLiqqSQD5G9V7fjObKWm2pDclTXX3PmnwD4Sk4/JuDkBxqr7nH2JmEyWtkHSdu+8yq+nyYZlZt6Tu+toDUJSajvxmNk6DwX/E3Z/OFveb2bSsPk3SwEjruvsSd+909848GgaQj6rht8FD/IOSNrj73cNKKyV1Zfe7JD2bf3sAilLLaf/Zkv4g6X0zezdb1iNpkaQnzWyhpM2S5hfT4tg3f376pTn++OOT9SuvvDLPdkal2tu7Rr4SvmfPnmT96quvrnvbqK5q+N39NUmV/gf8Nt92ADQLV/gBQRF+ICjCDwRF+IGgCD8QFOEHgqr58l7U79hjjy27hcKsWLEiWb/jjjsq1gYGRrwo9Efbt2+vqyfUhiM/EBThB4Ii/EBQhB8IivADQRF+ICjCDwTFFN1NUO3nr88///xk/YorrkjWp0+fXrH29ddfJ9et5r777kvWX3311WR97969De0fo8cU3QCSCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMb5gYMM4/wAkgg/EBThB4Ii/EBQhB8IivADQRF+IKiq4TezNjN72cw2mNkHZvbnbPltZvaFmb2b/ft98e0CyEvVi3zMbJqkae7+tplNkvSWpIskXSppj7v/veadcZEPULhaL/KpOmOPu/dJ6svu7zazDZJmNNYegLKN6j2/mc2UNFvSm9mia83sPTPrNbPJFdbpNrN1ZrauoU4B5Krma/vNbKKkf0u6092fNrOpknZIckl3aPCtwZVVtsFpP1CwWk/7awq/mY2T9Jyk1e5+9wj1mZKec/dTq2yH8AMFy+2LPWZmkh6UtGF48LMPAodcLGn9aJsEUJ5aPu0/R9Krkt6XtC9b3CPpckmna/C0f5Okq7IPB1Pb4sgPFCzX0/68EH6geHyfH0AS4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+IKiqP+CZsx2S/jvs8ZRsWStq1d5atS+J3uqVZ28n1frEpn6f/yc7N1vn7p2lNZDQqr21al8SvdWrrN447QeCIvxAUGWHf0nJ+09p1d5atS+J3upVSm+lvucHUJ6yj/wASlJK+M3sQjP72Mw+NbOby+ihEjPbZGbvZzMPlzrFWDYN2oCZrR+27Bgze9HMPsluR5wmraTeWmLm5sTM0qW+dq0243XTT/vN7FBJGyXNkbRV0lpJl7v7h01tpAIz2ySp091LHxM2s99I2iPpoaHZkMzsb5J2uvui7A/nZHf/a4v0dptGOXNzQb1Vmln6jyrxtctzxus8lHHkP1PSp+7+ubt/L+lxSfNK6KPlufsrknYesHiepGXZ/WUa/M/TdBV6awnu3ufub2f3d0samlm61Ncu0Vcpygj/DElbhj3eqtaa8tslrTGzt8ysu+xmRjB1aGak7Pa4kvs5UNWZm5vpgJmlW+a1q2fG67yVEf6RZhNppSGHs939V5J+J+ma7PQWtVksqV2D07j1SbqrzGaymaVXSLrO3XeV2ctwI/RVyutWRvi3Smob9vgESdtK6GNE7r4tux2Q9IwG36a0kv6hSVKz24GS+/mRu/e7+w/uvk/SAyrxtctmll4h6RF3fzpbXPprN1JfZb1uZYR/raRTzOxkMztc0mWSVpbQx0+Y2ZHZBzEysyMlXaDWm314paSu7H6XpGdL7GU/rTJzc6WZpVXya9dqM16XcpFPNpRxr6RDJfW6+51Nb2IEZvYzDR7tpcFvPD5aZm9m9pikczX4ra9+SbdK+qekJyWdKGmzpPnu3vQP3ir0dq5GOXNzQb1Vmln6TZX42uU543Uu/XCFHxATV/gBQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwjq/03vB3CtDy8wAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(xb[0].view(28,28))\n",
    "yb[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model,opt = get_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit():\n",
    "    for epoch in range(epochs):\n",
    "        for xb,yb in train_dl:\n",
    "            pred = model(xb)\n",
    "            loss = loss_func(pred, yb)\n",
    "            loss.backward()\n",
    "            opt.step()\n",
    "            opt.zero_grad()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.0762, grad_fn=<NllLossBackward>), tensor(0.9844))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss,acc = loss_func(model(xb), yb), accuracy(model(xb), yb)\n",
    "assert acc>0.7\n",
    "loss,acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want our training set to be in a random order, and that order should differ each iteration. But the validation set shouldn't be randomized."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 9 video](https://course19.fast.ai/videos/?lesson=9&t=3942)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Sampler():\n",
    "    def __init__(self, ds, bs, shuffle=False):\n",
    "        self.n,self.bs,self.shuffle = len(ds),bs,shuffle\n",
    "        \n",
    "    def __iter__(self):\n",
    "        self.idxs = torch.randperm(self.n) if self.shuffle else torch.arange(self.n)\n",
    "        for i in range(0, self.n, self.bs): yield self.idxs[i:i+self.bs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "small_ds = Dataset(*train_ds[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[tensor([0, 1, 2]), tensor([3, 4, 5]), tensor([6, 7, 8]), tensor([9])]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = Sampler(small_ds,3,False)\n",
    "[o for o in s]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[tensor([8, 0, 3]), tensor([5, 1, 2]), tensor([7, 6, 4]), tensor([9])]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = Sampler(small_ds,3,True)\n",
    "[o for o in s]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def collate(b):\n",
    "    xs,ys = zip(*b)\n",
    "    return torch.stack(xs),torch.stack(ys)\n",
    "\n",
    "class DataLoader():\n",
    "    def __init__(self, ds, sampler, collate_fn=collate):\n",
    "        self.ds,self.sampler,self.collate_fn = ds,sampler,collate_fn\n",
    "        \n",
    "    def __iter__(self):\n",
    "        for s in self.sampler: yield self.collate_fn([self.ds[i] for i in s])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_samp = Sampler(train_ds, bs, shuffle=True)\n",
    "valid_samp = Sampler(valid_ds, bs, shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dl = DataLoader(train_ds, sampler=train_samp, collate_fn=collate)\n",
    "valid_dl = DataLoader(valid_ds, sampler=valid_samp, collate_fn=collate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(3)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADXNJREFUeJzt3X+IVXUax/HPUxlpSlmSqU3ZjrHsEk1uQ2wUS7UY7SJYgVHQMmuyU1Cw1RYbQ1AUgSzbj6U/DKNBo99pbVK2GhHbryW0H5Rl2Q9cNceZNSOVijCf/WPOxGhzv/fOvefcc8fn/QK5957nnnMeLn7mnHu/596vubsAxHNI2Q0AKAfhB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8Q1GHN3JmZcTkhUDB3t1qe19CR38wuNLOPzexTM7u5kW0BaC6r99p+MztU0kZJcyRtlbRW0uXu/mFiHY78QMGaceQ/U9Kn7v65u38v6XFJ8xrYHoAmaiT8MyRtGfZ4a7ZsP2bWbWbrzGxdA/sCkLNGPvAb6dTiJ6f17r5E0hKJ036glTRy5N8qqW3Y4xMkbWusHQDN0kj410o6xcxONrPDJV0maWU+bQEoWt2n/e6+18yulbRa0qGSet39g9w6A1Couof66toZ7/mBwjXlIh8AYxfhB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0E1dYpu1KejoyNZv/766yvW2tvbk+tOmDAhWe/p6UnWjzrqqGT9hRdeqFjbvXt3cl0UiyM/EBThB4Ii/EBQhB8IivADQRF+ICjCDwTV0Cy9ZrZJ0m5JP0ja6+6dVZ7PLL0jmDhxYrK+efPmZP3oo4/Os51cffHFFxVrqesTJGn58uV5txNCrbP05nGRz3nuviOH7QBoIk77gaAaDb9LWmNmb5lZdx4NAWiORk/7z3b3bWZ2nKQXzewjd39l+BOyPwr8YQBaTENHfnfflt0OSHpG0pkjPGeJu3dW+zAQQHPVHX4zO9LMJg3dl3SBpPV5NQagWI2c9k+V9IyZDW3nUXf/Vy5dAShcQ+P8o94Z4/wjmjRpUrK+atWqZP3LL7+sWHvnnXeS686ePTtZP+mkk5L1tra2ZH38+PEVa/39/cl1zzrrrGS92vpR1TrOz1AfEBThB4Ii/EBQhB8IivADQRF+ICiG+tCQKVOmJOs33XRTXTVJWrBgQbK+bNmyZD0qhvoAJBF+ICjCDwRF+IGgCD8QFOEHgiL8QFBM0Y2G7NiR/uHm119/vWKt2jh/ta8bM87fGI78QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4/xoyOTJk5P1np6eurc9ffr0utdFdRz5gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiCoqr/bb2a9kuZKGnD3U7Nlx0h6QtJMSZskXeruX1XdGb/bP+Z0dHQk60899VSyPmvWrIq1jRs3JtedM2dOsr5ly5ZkPao8f7d/qaQLD1h2s6SX3P0USS9ljwGMIVXD7+6vSNp5wOJ5koZ+RmWZpIty7gtAwep9zz/V3fskKbs9Lr+WADRD4df2m1m3pO6i9wNgdOo98veb2TRJym4HKj3R3Ze4e6e7d9a5LwAFqDf8KyV1Zfe7JD2bTzsAmqVq+M3sMUn/kfRzM9tqZgslLZI0x8w+kTQnewxgDKk6zp/rzhjnbzldXV3J+u23356st7W1JevffvttxdrcuXOT67788svJOkaW5zg/gIMQ4QeCIvxAUIQfCIrwA0ERfiAofrr7IDBx4sSKtRtvvDG57i233JKsH3JI+viwc+eB3/na3znnnFOx9tFHHyXXRbE48gNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIzzHwSWLl1asXbJJZc0tO3ly5cn6/fee2+yzlh+6+LIDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBMc5/EGhvby9s24sXL07W33jjjcL2jWJx5AeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoKqO85tZr6S5kgbc/dRs2W2S/iTpf9nTetx9VVFNIm3NmjUVax0dHYVtW6p+HcCiRYsq1rZt21ZXT8hHLUf+pZIuHGH5Pe5+evaP4ANjTNXwu/srktLTsgAYcxp5z3+tmb1nZr1mNjm3jgA0Rb3hXyypXdLpkvok3VXpiWbWbWbrzGxdnfsCUIC6wu/u/e7+g7vvk/SApDMTz13i7p3u3llvkwDyV1f4zWzasIcXS1qfTzsAmqWWob7HJJ0raYqZbZV0q6Rzzex0SS5pk6SrCuwRQAHM3Zu3M7Pm7SyQ8ePHV6w9/PDDyXXPOOOMZP3EE0+sq6ch27dvr1hbsGBBct3Vq1c3tO+o3N1qeR5X+AFBEX4gKMIPBEX4gaAIPxAU4QeCYqjvIHfEEUck64cdlr7UY9euXXm2s5/vvvsuWb/hhhuS9fvvvz/Pdg4aDPUBSCL8QFCEHwiK8ANBEX4gKMIPBEX4gaAY50fSaaedlqzfc889yfp5551X9743b96crM+cObPubR/MGOcHkET4gaAIPxAU4QeCIvxAUIQfCIrwA0Exzt8CJkyYkKx/8803Tepk9CZPTk/T2NvbW7E2b968hvY9Y8aMZL2vr6+h7Y9VjPMDSCL8QFCEHwiK8ANBEX4gKMIPBEX4gaDSP9ouyczaJD0k6XhJ+yQtcfd/mNkxkp6QNFPSJkmXuvtXxbU6drW3tyfrr732WrL+/PPPJ+vr16+vWKs21r1w4cJkfdy4ccl6tbH2WbNmJespn332WbIedRw/L7Uc+fdK+ou7/0LSryVdY2a/lHSzpJfc/RRJL2WPAYwRVcPv7n3u/nZ2f7ekDZJmSJonaVn2tGWSLiqqSQD5G9V7fjObKWm2pDclTXX3PmnwD4Sk4/JuDkBxqr7nH2JmEyWtkHSdu+8yq+nyYZlZt6Tu+toDUJSajvxmNk6DwX/E3Z/OFveb2bSsPk3SwEjruvsSd+909848GgaQj6rht8FD/IOSNrj73cNKKyV1Zfe7JD2bf3sAilLLaf/Zkv4g6X0zezdb1iNpkaQnzWyhpM2S5hfT4tg3f376pTn++OOT9SuvvDLPdkal2tu7Rr4SvmfPnmT96quvrnvbqK5q+N39NUmV/gf8Nt92ADQLV/gBQRF+ICjCDwRF+IGgCD8QFOEHgqr58l7U79hjjy27hcKsWLEiWb/jjjsq1gYGRrwo9Efbt2+vqyfUhiM/EBThB4Ii/EBQhB8IivADQRF+ICjCDwTFFN1NUO3nr88///xk/YorrkjWp0+fXrH29ddfJ9et5r777kvWX3311WR97969De0fo8cU3QCSCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMb5gYMM4/wAkgg/EBThB4Ii/EBQhB8IivADQRF+IKiq4TezNjN72cw2mNkHZvbnbPltZvaFmb2b/ft98e0CyEvVi3zMbJqkae7+tplNkvSWpIskXSppj7v/veadcZEPULhaL/KpOmOPu/dJ6svu7zazDZJmNNYegLKN6j2/mc2UNFvSm9mia83sPTPrNbPJFdbpNrN1ZrauoU4B5Krma/vNbKKkf0u6092fNrOpknZIckl3aPCtwZVVtsFpP1CwWk/7awq/mY2T9Jyk1e5+9wj1mZKec/dTq2yH8AMFy+2LPWZmkh6UtGF48LMPAodcLGn9aJsEUJ5aPu0/R9Krkt6XtC9b3CPpckmna/C0f5Okq7IPB1Pb4sgPFCzX0/68EH6geHyfH0AS4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+IKiqP+CZsx2S/jvs8ZRsWStq1d5atS+J3uqVZ28n1frEpn6f/yc7N1vn7p2lNZDQqr21al8SvdWrrN447QeCIvxAUGWHf0nJ+09p1d5atS+J3upVSm+lvucHUJ6yj/wASlJK+M3sQjP72Mw+NbOby+ihEjPbZGbvZzMPlzrFWDYN2oCZrR+27Bgze9HMPsluR5wmraTeWmLm5sTM0qW+dq0243XTT/vN7FBJGyXNkbRV0lpJl7v7h01tpAIz2ySp091LHxM2s99I2iPpoaHZkMzsb5J2uvui7A/nZHf/a4v0dptGOXNzQb1Vmln6jyrxtctzxus8lHHkP1PSp+7+ubt/L+lxSfNK6KPlufsrknYesHiepGXZ/WUa/M/TdBV6awnu3ufub2f3d0samlm61Ncu0Vcpygj/DElbhj3eqtaa8tslrTGzt8ysu+xmRjB1aGak7Pa4kvs5UNWZm5vpgJmlW+a1q2fG67yVEf6RZhNppSGHs939V5J+J+ma7PQWtVksqV2D07j1SbqrzGaymaVXSLrO3XeV2ctwI/RVyutWRvi3Smob9vgESdtK6GNE7r4tux2Q9IwG36a0kv6hSVKz24GS+/mRu/e7+w/uvk/SAyrxtctmll4h6RF3fzpbXPprN1JfZb1uZYR/raRTzOxkMztc0mWSVpbQx0+Y2ZHZBzEysyMlXaDWm314paSu7H6XpGdL7GU/rTJzc6WZpVXya9dqM16XcpFPNpRxr6RDJfW6+51Nb2IEZvYzDR7tpcFvPD5aZm9m9pikczX4ra9+SbdK+qekJyWdKGmzpPnu3vQP3ir0dq5GOXNzQb1Vmln6TZX42uU543Uu/XCFHxATV/gBQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwjq/03vB3CtDy8wAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xb,yb = next(iter(valid_dl))\n",
    "plt.imshow(xb[0].view(28,28))\n",
    "yb[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAC/xJREFUeJzt3V+IXPUZxvHn6epeGL0waJIlmmpFS0WoliUUUkpKidhS2ORCMWhJqWS9UGigFw3eKBRBS7XtlbDBYMR/FdQmEalKKE0LUYwiGhM1KlHTDRs1FRXRxeTtxZ6UNe6cmcycM2c27/cDy8yc3/nzMskzvzNz/vwcEQKQz7eaLgBAMwg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkTuvnxmxzOiFQs4hwJ/P11PPbvsr2G7bfsr2xl3UB6C93e26/7SFJb0paJemgpBckrY2IvSXL0PMDNetHz79c0lsR8U5ETEt6RNJYD+sD0Ee9hH+ppPdnvT5YTPsa2+O2d9ve3cO2AFSslx/85tq1+MZufURMSJqQ2O0HBkkvPf9BSefPen2epMneygHQL72E/wVJF9u+0PawpGslbaumLAB163q3PyK+sn2zpKclDUnaHBGvVVYZgFp1faivq43xnR+oXV9O8gEwfxF+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kFRfh+hGPU47rfU/465du0qX3bu35biqkqT169eXtk9PT5e2Y3DR8wNJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUj2N0mv7gKRPJR2V9FVEjLaZn1F6azA8PNyy7Ysvvuhp3WNjY6Xt27dv72n9qF6no/RWcZLPTyLiwwrWA6CP2O0Hkuo1/CHpGdsv2h6voiAA/dHrbv+KiJi0vUjSs7Zfj4ids2coPhT4YAAGTE89f0RMFo+HJT0hafkc80xExGi7HwMB9FfX4be9wPZZx59LulLSnqoKA1CvXnb7F0t6wvbx9TwUEX+vpCoAtes6/BHxjqTvV1gLBtDq1atL2znOP39xqA9IivADSRF+ICnCDyRF+IGkCD+QFLfuRqlLLrmk6RJQE3p+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK4/wode655/bU/sEHH1RZDipEzw8kRfiBpAg/kBThB5Ii/EBShB9IivADSXGcH6XaXc/frp3j/IOLnh9IivADSRF+ICnCDyRF+IGkCD+QFOEHkmobftubbR+2vWfWtIW2n7W9v3g8u94yMaiGhoZK/zC4Oun575N01QnTNkraEREXS9pRvAYwj7QNf0TslHTkhMljkrYUz7dIWl1xXQBq1u13/sURcUiSisdF1ZUEoB9qP7ff9rik8bq3A+DkdNvzT9kekaTi8XCrGSNiIiJGI2K0y20BqEG34d8maV3xfJ2krdWUA6BfOjnU97CkXZK+a/ug7Rsk3SFple39klYVrwHMI46I/m3M7t/GEhkeHm7Z9tFHH5Uuu2DBgp62vX379tL2sbGxntaPkxcR7mQ+zvADkiL8QFKEH0iK8ANJEX4gKcIPJMWtu08B09PTLds2bdpUuuyGDRuqLgfzBD0/kBThB5Ii/EBShB9IivADSRF+ICnCDyTFcX6UssuvDl24cGFp+xlnnNGy7fPPP++qJlSDnh9IivADSRF+ICnCDyRF+IGkCD+QFOEHkuI4/ynu7bff7mn5drd2X7FiRWn7smXLWra9/vrrXdWEatDzA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBSbYfotr1Z0i8kHY6Iy4ppt0laL+mDYrZbIuKpthtjiO6+W7JkSWn75ORkaXu76/nb/f+59NJLW7ZxnL8eVQ7RfZ+kq+aY/qeIuLz4axt8AIOlbfgjYqekI32oBUAf9fKd/2bbr9jebPvsyioC0Bfdhv8eSRdJulzSIUl3tZrR9rjt3bZ3d7ktADXoKvwRMRURRyPimKRNkpaXzDsREaMRMdptkQCq11X4bY/MerlG0p5qygHQL20v6bX9sKSVks6xfVDSrZJW2r5cUkg6IOnGGmsEUIO24Y+ItXNMvreGWjCA2h3Hx/zFGX5AUoQfSIrwA0kRfiApwg8kRfiBpLh19yluamqqtP2BBx4obb/++uurLAcDhJ4fSIrwA0kRfiApwg8kRfiBpAg/kBThB5LiOP8prt0luceOHStt7/XW3SMjIy3b9u/fX7rs0aNHS9vRG3p+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK4/wo1eutu3fs2NGybc2aNaXLbt26tadtoxw9P5AU4QeSIvxAUoQfSIrwA0kRfiApwg8k1fY4v+3zJd0vaYmkY5ImIuIvthdK+qukCyQdkHRNRPy3vlJxqnnuueeaLiG1Tnr+ryT9NiK+J+mHkm6yfamkjZJ2RMTFknYUrwHME23DHxGHIuKl4vmnkvZJWippTNKWYrYtklbXVSSA6p3Ud37bF0i6QtLzkhZHxCFp5gNC0qKqiwNQn47P7bd9pqTHJG2IiE/a3dtt1nLjksa7Kw9AXTrq+W2frpngPxgRjxeTp2yPFO0jkg7PtWxETETEaESMVlEwgGq0Db9nuvh7Je2LiLtnNW2TtK54vk4Sl2AB80gnu/0rJP1S0qu2Xy6m3SLpDkmP2r5B0nuSrq6nRMxn7777bsu2L7/8so+V4ERtwx8R/5bU6gv+T6stB0C/cIYfkBThB5Ii/EBShB9IivADSRF+IClu3Y1a3XnnnS3bPv744z5WghPR8wNJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUhznT+6pp54qbR8aGiptv+6660rbuT334KLnB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkHBH925jdv40BSUVER2Pp0fMDSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFJtw2/7fNv/sL3P9mu2f1NMv832f2y/XPz9vP5yAVSl7Uk+tkckjUTES7bPkvSipNWSrpH0WUT8seONcZIPULtOT/JpeyefiDgk6VDx/FPb+yQt7a08AE07qe/8ti+QdIWk54tJN9t+xfZm22e3WGbc9m7bu3uqFEClOj633/aZkv4p6faIeNz2YkkfSgpJv9fMV4Nft1kHu/1AzTrd7e8o/LZPl/SkpKcj4u452i+Q9GREXNZmPYQfqFllF/bYtqR7Je2bHfzih8Dj1kjac7JFAmhOJ7/2/0jSvyS9KulYMfkWSWslXa6Z3f4Dkm4sfhwsWxc9P1CzSnf7q0L4gfpxPT+AUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGk2t7As2IfSnp31utzimmDaFBrG9S6JGrrVpW1fbvTGft6Pf83Nm7vjojRxgooMai1DWpdErV1q6na2O0HkiL8QFJNh3+i4e2XGdTaBrUuidq61UhtjX7nB9Ccpnt+AA1pJPy2r7L9hu23bG9sooZWbB+w/Wox8nCjQ4wVw6Adtr1n1rSFtp+1vb94nHOYtIZqG4iRm0tGlm70vRu0Ea/7vttve0jSm5JWSToo6QVJayNib18LacH2AUmjEdH4MWHbP5b0maT7j4+GZPsPko5ExB3FB+fZEfG7AantNp3kyM011dZqZOlfqcH3rsoRr6vQRM+/XNJbEfFORExLekTSWAN1DLyI2CnpyAmTxyRtKZ5v0cx/nr5rUdtAiIhDEfFS8fxTScdHlm70vSupqxFNhH+ppPdnvT6owRryOyQ9Y/tF2+NNFzOHxcdHRioeFzVcz4najtzcTyeMLD0w7103I15XrYnwzzWayCAdclgRET+Q9DNJNxW7t+jMPZIu0swwbock3dVkMcXI0o9J2hARnzRZy2xz1NXI+9ZE+A9KOn/W6/MkTTZQx5wiYrJ4PCzpCc18TRkkU8cHSS0eDzdcz/9FxFREHI2IY5I2qcH3rhhZ+jFJD0bE48Xkxt+7uepq6n1rIvwvSLrY9oW2hyVdK2lbA3V8g+0FxQ8xsr1A0pUavNGHt0laVzxfJ2lrg7V8zaCM3NxqZGk1/N4N2ojXjZzkUxzK+LOkIUmbI+L2vhcxB9vf0UxvL81c8fhQk7XZfljSSs1c9TUl6VZJf5P0qKRlkt6TdHVE9P2Htxa1rdRJjtxcU22tRpZ+Xg2+d1WOeF1JPZzhB+TEGX5AUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5L6H9frjr56YmyIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xb,yb = next(iter(train_dl))\n",
    "plt.imshow(xb[0].view(28,28))\n",
    "yb[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(9)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADOZJREFUeJzt3V+oXfWZxvHnMW1E0ypq0EnsUWvUMoNiqieHAQfJUCzOmBB7EW3AkpE6pxcNTLEXowGpNwM69s/MVfGUhKahsS20GXNRZhp1wCmUkhhjTfOvscTmH4k11RpvgvrOxVkpp8nZv73de629dvp+PxDO3uvda62XTZ6z1j6/tdfPESEA+VzQdgMA2kH4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8k9ZFh7sw2lxMCDYsI9/K6gY78tu+2vc/2AduPDLItAMPlfq/ttz1H0n5Jd0k6LGmbpFURsbuwDkd+oGHDOPJPSDoQEb+NiNOSfiBpxQDbAzBEg4T/akmHZjw/XC37M7YnbW+3vX2AfQGo2SB/8Jvt1OKc0/qImJI0JXHaD4ySQY78hyWNzXj+CUlHB2sHwLAMEv5tkm60/UnbcyV9XtKWetoC0LS+T/sj4j3bayT9j6Q5ktZHxK9r6wxAo/oe6utrZ3zmBxo3lIt8AJy/CD+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iq7ym6Jcn2QUnvSHpf0nsRMV5HUwCaN1D4K38fEb+vYTsAhojTfiCpQcMfkn5m+yXbk3U0BGA4Bj3tvyMijtq+UtJW23sj4sWZL6h+KfCLARgxjoh6NmQ/LulURHy98Jp6dgago4hwL6/r+7Tf9jzbHz/zWNJnJe3qd3sAhmuQ0/6rJG22fWY7myLiv2vpCkDjajvt72lnnPYDjWv8tB/A+Y3wA0kRfiApwg8kRfiBpAg/kFQd3+pL78EHHyzW161bN6ROzlVdh9HR3r17i/WtW7cOtP+pqamOtV27uCasTRz5gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApxvlr8MADDxTrw/za9Ifd90033TRQvZuLL764Y+2hhx4aaNsYDEd+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iKW3f3qPS9+BdeeKG47p133lms7969u1hfvnx5sb5kyZKOtYmJieK63axZs6ZYnzt3brF+6tSpjrWFCxcW13333XeLdcyOW3cDKCL8QFKEH0iK8ANJEX4gKcIPJEX4gaS6jvPbXi9pmaQTEXFztexyST+UdJ2kg5Lui4g/dN3ZeTzOf8MNN3Ss7dixo7juvHnzivWdO3cW67fffnux3qSTJ08W65deemnf2167dm2x/uSTT/a97czqHOf/rqS7z1r2iKTnI+JGSc9XzwGcR7qGPyJelHT2r/8VkjZUjzdIurfmvgA0rN/P/FdFxDFJqn5eWV9LAIah8Xv42Z6UNNn0fgB8OP0e+Y/bXiBJ1c8TnV4YEVMRMR4R433uC0AD+g3/Fkmrq8erJT1bTzsAhqVr+G0/I+kXkj5l+7DtL0p6QtJdtn8j6a7qOYDzSNfP/BGxqkPpMzX3MtIOHDjQsbZt27biukuXLi3Wu42VX3HFFcX6m2++WawP4rXXXivWb7vttr63ff311/e9LgbHFX5AUoQfSIrwA0kRfiApwg8kRfiBpLh1dw1Wreo0Gjpt48aNxXrptuCS9PTTTxfrjz76aMfa22+/XVz3oosuKtb3799frHe7/XbJkSNHivVrrrmm721nxq27ARQRfiApwg8kRfiBpAg/kBThB5Ii/EBSjPMPwVNPPVWsP/zwwwNt/5VXXulYe/nll4vrLlu2rFifP39+Xz31gnH+ZjDOD6CI8ANJEX4gKcIPJEX4gaQIP5AU4QeSYpx/CObMmVOsL1++vFi///77i/UVK1Z0rF144YXFdbvZvHlzsT4+Xp6IaWxsrGONcf5mMM4PoIjwA0kRfiApwg8kRfiBpAg/kBThB5LqOs5ve72kZZJORMTN1bLHJf2zpDeql62NiJ923VnScf6mlabwvuCCwX6/d5v+e9OmTcX6ypUrO9aOHj1aXHfRokXF+unTp4v1rOoc5/+upLtnWf6tiFhc/esafACjpWv4I+JFSSeH0AuAIRrknHCN7V/ZXm/7sto6AjAU/Yb/25IWSVos6Zikb3R6oe1J29ttb+9zXwAa0Ff4I+J4RLwfER9I+o6kicJrpyJiPCLK3wABMFR9hd/2ghlPPydpVz3tABiWj3R7ge1nJC2VNN/2YUlfk7TU9mJJIemgpC812COABnQNf0TMNvn8ugZ6QZ+6jcWPqoULFxbr99xzT7He7V4DKOMKPyApwg8kRfiBpAg/kBThB5Ii/EBSXYf6gJK9e/e23QL6xJEfSIrwA0kRfiApwg8kRfiBpAg/kBThB5JinB8Dee6554r1xx57rO9tT0x0vEGUJL7SOyiO/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOP8GMi+ffuK9ddff71j7dprry2uu2TJkr56Qm848gNJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUl3H+W2PSfqepL+S9IGkqYj4T9uXS/qhpOskHZR0X0T8oblWMYreeOONYv2tt97qWOs2zo9m9XLkf0/SVyPiryX9raQv2/4bSY9Iej4ibpT0fPUcwHmia/gj4lhE7KgevyNpj6SrJa2QtKF62QZJ9zbVJID6fajP/Lavk/RpSb+UdFVEHJOmf0FIurLu5gA0p+dr+21/TNKPJX0lIv5ou9f1JiVN9tcegKb0dOS3/VFNB//7EfGTavFx2wuq+gJJJ2ZbNyKmImI8IsbraBhAPbqG39OH+HWS9kTEN2eUtkhaXT1eLenZ+tsD0JReTvvvkPQFSa/a3lktWyvpCUk/sv1FSb+TtLKZFgE0oWv4I+Lnkjp9wP9Mve0AGBau8AOSIvxAUoQfSIrwA0kRfiApwg8kxa27MbJuueWWYn1sbKxYP3ToUJ3t/MXhyA8kRfiBpAg/kBThB5Ii/EBShB9IivADSTHOj5E1f/78Yv2SSy4ZUid/mTjyA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFJdw297zPb/2t5j+9e2/6Va/rjtI7Z3Vv/+sfl2AdSll5t5vCfpqxGxw/bHJb1ke2tV+1ZEfL259gA0pWv4I+KYpGPV43ds75F0ddONAWjWh/rMb/s6SZ+W9Mtq0Rrbv7K93vZlHdaZtL3d9vaBOgVQq57Db/tjkn4s6SsR8UdJ35a0SNJiTZ8ZfGO29SJiKiLGI2K8hn4B1KSn8Nv+qKaD//2I+IkkRcTxiHg/Ij6Q9B1JE821CaBuvfy135LWSdoTEd+csXzBjJd9TtKu+tsD0JRe/tp/h6QvSHrV9s5q2VpJq2wvlhSSDkr6UiMd4ry2cePGjrVbb721uO6RI0eK9ZMnT/bVE6b18tf+n0vyLKWf1t8OgGHhCj8gKcIPJEX4gaQIP5AU4QeSIvxAUo6I4e3MHt7OgKQiYrah+XNw5AeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpHr5Pn+dfi/p9RnP51fLRtGo9jaqfUn01q86e7u21xcO9SKfc3Zubx/Ve/uNam+j2pdEb/1qqzdO+4GkCD+QVNvhn2p5/yWj2tuo9iXRW79a6a3Vz/wA2tP2kR9AS1oJv+27be+zfcD2I2300Intg7ZfrWYebnWKsWoatBO2d81YdrntrbZ/U/2cdZq0lnobiZmbCzNLt/rejdqM10M/7bc9R9J+SXdJOixpm6RVEbF7qI10YPugpPGIaH1M2Padkk5J+l5E3Fwt+3dJJyPiieoX52UR8a8j0tvjkk61PXNzNaHMgpkzS0u6V9I/qcX3rtDXfWrhfWvjyD8h6UBE/DYiTkv6gaQVLfQx8iLiRUlnz0yxQtKG6vEGTf/nGboOvY2EiDgWETuqx+9IOjOzdKvvXaGvVrQR/qslHZrx/LBGa8rvkPQz2y/Znmy7mVlcVU2bfmb69Ctb7udsXWduHqazZpYemfeunxmv69ZG+Ge7xdAoDTncERG3SfoHSV+uTm/Rm55mbh6WWWaWHgn9znhdtzbCf1jS2Iznn5B0tIU+ZhURR6ufJyRt1ujNPnz8zCSp1c8TLffzJ6M0c/NsM0trBN67UZrxuo3wb5N0o+1P2p4r6fOStrTQxzlsz6v+ECPb8yR9VqM3+/AWSaurx6slPdtiL39mVGZu7jSztFp+70ZtxutWLvKphjL+Q9IcSesj4t+G3sQsbF+v6aO9NP2Nx01t9mb7GUlLNf2tr+OSvibpvyT9SNI1kn4naWVEDP0Pbx16W6rpU9c/zdx85jP2kHv7O0n/J+lVSR9Ui9dq+vN1a+9doa9VauF94wo/ICmu8AOSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kNT/AyvcvaDYLWD7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xb,yb = next(iter(train_dl))\n",
    "plt.imshow(xb[0].view(28,28))\n",
    "yb[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.2939, grad_fn=<NllLossBackward>), tensor(0.9375))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model,opt = get_model()\n",
    "fit()\n",
    "\n",
    "loss,acc = loss_func(model(xb), yb), accuracy(model(xb), yb)\n",
    "assert acc>0.7\n",
    "loss,acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyTorch DataLoader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 9 video](https://course19.fast.ai/videos/?lesson=9&t=4171)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "from torch.utils.data import DataLoader, SequentialSampler, RandomSampler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dl = DataLoader(train_ds, bs, sampler=RandomSampler(train_ds), collate_fn=collate)\n",
    "valid_dl = DataLoader(valid_ds, bs, sampler=SequentialSampler(valid_ds), collate_fn=collate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.2196, grad_fn=<NllLossBackward>), tensor(0.9375))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model,opt = get_model()\n",
    "fit()\n",
    "loss_func(model(xb), yb), accuracy(model(xb), yb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch's defaults work fine for most things however:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dl = DataLoader(train_ds, bs, shuffle=True, drop_last=True)\n",
    "valid_dl = DataLoader(valid_ds, bs, shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.1029, grad_fn=<NllLossBackward>), tensor(0.9688))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model,opt = get_model()\n",
    "fit()\n",
    "\n",
    "loss,acc = loss_func(model(xb), yb), accuracy(model(xb), yb)\n",
    "assert acc>0.7\n",
    "loss,acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that PyTorch's `DataLoader`, if you pass `num_workers`, will use multiple threads to call your `Dataset`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You **always** should also have a [validation set](http://www.fast.ai/2017/11/13/validation-sets/), in order to identify if you are overfitting.\n",
    "\n",
    "We will calculate and print the validation loss at the end of each epoch.\n",
    "\n",
    "(Note that we always call `model.train()` before training, and `model.eval()` before inference, because these are used by layers such as `nn.BatchNorm2d` and `nn.Dropout` to ensure appropriate behaviour for these different phases.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 9 video](https://course19.fast.ai/videos/?lesson=9&t=4260)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit(epochs, model, loss_func, opt, train_dl, valid_dl):\n",
    "    for epoch in range(epochs):\n",
    "        # Handle batchnorm / dropout\n",
    "        model.train()\n",
    "#         print(model.training)\n",
    "        for xb,yb in train_dl:\n",
    "            loss = loss_func(model(xb), yb)\n",
    "            loss.backward()\n",
    "            opt.step()\n",
    "            opt.zero_grad()\n",
    "\n",
    "        model.eval()\n",
    "#         print(model.training)\n",
    "        with torch.no_grad():\n",
    "            tot_loss,tot_acc = 0.,0.\n",
    "            for xb,yb in valid_dl:\n",
    "                pred = model(xb)\n",
    "                tot_loss += loss_func(pred, yb)\n",
    "                tot_acc  += accuracy (pred,yb)\n",
    "        nv = len(valid_dl)\n",
    "        print(epoch, tot_loss/nv, tot_acc/nv)\n",
    "    return tot_loss/nv, tot_acc/nv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Question*: Are these validation results correct if batch size varies?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`get_dls` returns dataloaders for the training and validation sets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def get_dls(train_ds, valid_ds, bs, **kwargs):\n",
    "    return (DataLoader(train_ds, batch_size=bs, shuffle=True, **kwargs),\n",
    "            DataLoader(valid_ds, batch_size=bs*2, **kwargs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, our whole process of obtaining the data loaders and fitting the model can be run in 3 lines of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 tensor(0.1573) tensor(0.9533)\n",
      "1 tensor(0.5863) tensor(0.8684)\n",
      "2 tensor(0.1299) tensor(0.9609)\n",
      "3 tensor(0.1178) tensor(0.9664)\n",
      "4 tensor(0.1283) tensor(0.9625)\n"
     ]
    }
   ],
   "source": [
    "train_dl,valid_dl = get_dls(train_ds, valid_ds, bs)\n",
    "model,opt = get_model()\n",
    "loss,acc = fit(5, model, loss_func, opt, train_dl, valid_dl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert acc>0.9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Converted 03_minibatch_training.ipynb to nb_03.py\r\n"
     ]
    }
   ],
   "source": [
    "!python notebook2script.py 03_minibatch_training.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
